1. Technical Field
The disclosed embodiments relate in general to user interfaces of computing devices and, more specifically, to systems and methods for implementing and using gesture based user interface widgets with camera input.
2. Description of the Related Art
Gesture input is now a common way to interact with various computer systems. Examples of gesture-based user interfaces include touchscreens on phones and tablets, as well as in-the-air touchless gesture controllers for gaming systems. Research systems include gesture input for projected displays such as the IBM Everywhere Displays system described in Kjeldsen, R., C., Pingali, G., Hartman, J., Levas, T., Podlaseck, M. Interacting with steerable projected displays. Intl. Conf. on Automatic Face and Gesture Recognition (FGR '02), pp. 402-407 and Pinhanez, C., Kjeldsen, R., Tang, L., Levas, A., Podlaseck, M., Sukaviriya, N. and Pingali, G. Creating touch-screens anywhere with interactive projected displays. Proc. ACM Multimedia '03 (demo), pp. 460-461. More specialized applications include systems for enabling browsing of medical images such as the system described in Wachs, J., Stern, H., Edan, Y., Gillam, M., Handler, J., Feied, C., Smith, M. A gesture-based tool for sterile browsing of radiology images, J. American Medical Informatics Assoc., 15 (3) 321-323 (2008).
For projected displays and other systems that lack gesture support, it would be useful to enable such input capabilities in a low-cost and robust way. Often this is accomplished by using a video camera (e.g. a webcam) as an input device to track the user's finger or hand. In the case of the Everywhere Displays system, the objective of finger tracking is to detect touch events on a projected display. For the aforesaid system used in a medical setting where it is important to maintain sterile and hence not desirable for the user to touch the display, hand tracking is used to detect in-the-air touchless gestures.
The problems of employing a low-cost video camera for finger and hand tracking are the following. Because a single camera cannot provide depth information, the “touch gesture” must be identified indirectly from the gesture motion. In the Everywhere Displays system, described above, a touch event is defined to occur when the user's fingertip moves to the button and then away from the button. As would be appreciated by those of skill in the art, this does not work for common tasks like pressing the “Next” button repeatedly in a slideshow application, or pressing buttons on a virtual keypad without retracting the hand.
Another problem that was noticed in connection with the aforesaid Everywhere Displays system is that when users see a regular button, they tend to want to press on it (rather than make the “touch gesture”). This issue arises due to the lack of perceived affordance as the user cannot really figure out the correct way to interact with the button just by looking at it. A lack of visual cues is even more prevalent with the aforesaid in-the-air touchless gesture input systems.
A further issue complicating the operation of the conventional systems is the need for calibration of the user interface component for each user's hand or skin color, or the color of the glove worn by the user of the system, in order to enable reliable hand gesture recognition. The aforesaid calibration requires additional operations to be performed by the users decreasing system's overall usability.
Additional universal challenges facing conventional gesture based user interfaces include avoiding interpreting ordinary gestures and motions of the users that were not intended to constitute input for the computer system. On the other hand, the gestures must be easy and comfortable for the user to perform and, at the same time, be reliably detected by the system. To address these challenges, one exemplary system described in Wilson, A. D., Using a depth camera as a touch sensor, Proc. ITS '10, pp. 69-72 provides a robust way to enter and exit hand tracking states for supporting continuous gesture events. The user interaction style implemented by the aforesaid system is similar to glove input devices for virtual reality.
As would be appreciated by those of skill in the art, in view of the aforesaid deficiencies of the conventional technology, new and improved gesture based user interfaces with improved robustness of gesture detection are needed.